Papers with multilingual representations
Language Directions in Multilingual LLMs: A Layer-wise Diagnostic Study of Token Alignment and Pretraining Imprint (2026.acl-srw)
Copied to clipboard
| Challenge: | Using a unified probing framework, we analyze six multilingual LLMs across five languages. |
| Approach: | They analyze multilingual representations across five languages and analyze their behavior . they find that accuracy rises by +73.5 to +80.7 points from L0 to L1 on average . |
| Outcome: | The proposed framework enables a consistent and substantial early jump in accuracy across models . the token–language alignment measures where vocabulary sharing peaks . |
BERT is Not an Interlingua and the Bias of Tokenization (D19-61)
Copied to clipboard
| Challenge: | Cananical Correlation Analysis (CCA) of the internal representations of a pre- trained, multilingual BERT model reveals that the model partitions representations for each language rather than using a common, shared, interlingual space. |
| Approach: | They propose to use a multilingual BERT model to partition representations for each language rather than using a common, shared, interlingual space. |
| Outcome: | The results show that the model partitions representations for each language rather than using a common, shared, interlingual space. |
Multi-view Subword Regularization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing subword regularization methods for multilingual pretrained representations are suboptimal for multi-lingual transfer. |
| Approach: | They propose a method that enforces consistency between standard and probabilistic segmentations. |
| Outcome: | The proposed method improves the effectiveness of cross-lingual transfer by 2.5 points over standard methods. |
MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning (2021.naacl-main)
Copied to clipboard
Mengzhou Xia, Guoqing Zheng, Subhabrata Mukherjee, Milad Shokouhi, Graham Neubig, Ahmed Hassan Awadallah
| Challenge: | Recent work shows that multilingual representations are disjointed across languages, bringing additional challenges for transfer onto extremely low-resource languages. |
| Approach: | They propose a meta-learning based framework that learns to transform representations judiciously from auxiliary languages to a target one and brings their representation spaces closer for effective transfer. |
| Outcome: | The proposed framework learns to transform representations from auxiliary languages to a target language and brings their representation spaces closer for effective transfer. |
Improving Cross Lingual Transfer by Pretraining with Active Forgetting (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has shown that encoder-only LLMs show impressive cross lingual transfer of their capabilities from English to other languages. |
| Approach: | They propose a pretraining strategy that uses active forgetting to achieve similar cross lingual transfer in decoder-only LLMs. |
| Outcome: | The proposed model improves cross lingual transfer capabilities on non-English languages despite being trained on English data. |
Combining Static and Contextualised Multilingual Embeddings (2022.findings-acl)
Copied to clipboard
| Challenge: | Static embeddings are less expressive than contextual language models, but can be more straightforwardly aligned across multiple languages. |
| Approach: | They extract static embeddings for 40 languages from XLM-R and validate them with cross-lingual word retrieval and then align them using VecMap. |
| Outcome: | The proposed approach improves multilingual representations by leveraging static embeddings and a pre-training code. |
Assessing Multilingual Fairness in Pre-trained Multimodal Representations (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent pre-trained multimodal models have shown exceptional capabilities towards connecting images and natural language. |
| Approach: | They propose two new fairness notions for pre-trained multimodal models that consider language as the fairness recipient. |
| Outcome: | The proposed models can be generalized to multilingualism by cross-lingual alignment . the results show that the models are individually fair across languages . |
Do Multilingual Neural Machine Translation Models Contain Language Pair Specific Attention Heads? (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent studies on multilingual representations focus on whether there is an emergence of language-independent representations or whether multilingual models partition their weights among different languages. |
| Approach: | They analyze encoder self-attention and encoder-decoder attention heads in a multilingual neural translation model. |
| Outcome: | The proposed model is based on a multilingual neural translation model with a language-independent representation. |
Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer (2020.acl-main)
Copied to clipboard
| Challenge: | Multilingual word embeddings embed words from many languages into a single semantic space such that words with similar meanings are close to each other regardless of the language. |
| Approach: | They propose to use multilingual word embeddings to align embeddable words from multiple languages into a single semantic space so that words with similar meanings are close to each other regardless of the language. |
| Outcome: | The proposed model can be used to learn gender bias in multilingual representations and to improve transfer learning. |
Explicit Alignment Objectives for Multilingual Bidirectional Encoders (2021.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained cross-lingual encoders have proven impressively effective at enabling transfer-learning of NLP systems from high-resource languages to low-resourced languages. |
| Approach: | They propose a method to align multilingual encoders using two explicit alignment objectives that align the multilingual representations at different granularities. |
| Outcome: | The proposed method achieves gains of up to 1.1 average F1 score on sequence tagging and 27.3 average accuracy on retrieval over the XLM-R-large model. |
AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual large language models (LLMs) possess impressive multilingual understanding and generation capabilities, but performance and cross-lingual alignment often lag for non-dominant languages. |
| Approach: | They propose a representation-level framework to enhance multilingual performance of pre-trained LLMs by integrating multilingual semantic alignment and language feature integration. |
| Outcome: | The proposed framework improves multilingual capability of pre-trained LLMs by bringing representations closer and improving cross-lingual alignment. |
Learning Cross-Lingual IR from an English Retriever (2022.naacl-main)
Copied to clipboard
| Challenge: | DR.DECR is a cross-lingual information retrieval system trained using multi-stage knowledge distillation (KD) DRDECR demonstrates superior accuracy over direct fine-tuning with labeled CLIR data. |
| Approach: | They propose a cross-lingual information retrieval system with multi-stage knowledge distillation . they teach powerful multilingual representations and CLIR by optimizing two corresponding KD objectives . |
| Outcome: | The proposed system is the best single-model retriever on the XOR-TyDi benchmark . it is based on a multi-stage knowledge distillation process that can be expensive . |
On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment (2020.emnlp-main)
Copied to clipboard
| Challenge: | Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer) however, recent work has shown that this approach can degrade performance on high-resource languages, a phenomenon known as negative interference. |
| Approach: | They propose a meta-learning algorithm that adds language-specific parameters as meta-parameters and trains them in a manner that explicitly improves shared layers’ generalization on all languages. |
| Outcome: | The proposed model improves cross-lingual transferability and generalization on all languages, and improves on the language-specific parameters. |
Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel on English NLU tasks, yet struggle to extend their NLU capabilities to underrepresented languages. |
| Approach: | They integrate machine translation models (MT) directly into LLM backbones via sample-efficient self-distillation. |
| Outcome: | The proposed model outperforms translation-test models on 127 low-resource languages. |
LangBridge: Multilingual Reasoning Without Multilingual Supervision (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to adapt language models for multilingual reasoning tasks require multilingual supervision. |
| Approach: | They propose a zero-shot approach to adapt language models for multilingual reasoning tasks without multilingual supervision by bridging two models by introducing minimal trainable parameters between them. |
| Outcome: | The proposed approach significantly improves multilingual reasoning capabilities on low-resource languages. |
Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that contextual language models display outlier dimensions . this is true for monolingual and multilingual models, but little work has been done on multilingual contexts . |
| Approach: | They investigate outlier dimensions and their relationship to anisotropy in multilingual contexts . they focus on cross-lingual semantic similarity tasks . |
| Outcome: | The proposed model improves on cross-lingual semantic similarity tasks. |
How Multilingual is Multilingual BERT? (P19-1)
Copied to clipboard
| Challenge: | Existing studies have shown that deep, contextualized language models can encode syntactic and named entity information, but they have focused on what models trained on English capture about English. |
| Approach: | They propose a multilingual model pre-trained from monolingual Wikipedia corpora . they show that multilingual BERT is surprisingly good at zero-shot cross-lingual model transfer . |
| Outcome: | The proposed model can find translation pairs, but it exhibits systematic deficiencies affecting certain language pairs. |
MEVTR: A Multilingual Model Enhanced with Visual Text Representations (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models that generate multilingual text representations perform poorly on low-resource languages due to lack of representation space and model capacity. |
| Approach: | They propose a multilingual model enhanced with visual text representations which complements textual representations and extends multilingual representation space with visual representations. |
| Outcome: | The proposed model outperforms state-of-the-art models on zero-shot cross-lingual transfer tasks without the target language adapter. |
Discriminating Form and Meaning in Multilingual Models with Minimal-Pair ABX Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that multilingual models encode languagespecific information and language-agnostic features, but the nature and interaction of these representations is not fully understood. |
| Approach: | They propose a set of training-free ABX-style discrimination tasks to evaluate how multilingual language models represent language identity (form) and semantic content (meaning). |
| Outcome: | The proposed tasks show that language discrimination declines over training and strengthens over time and stabilizes in deeper layers. |